User Interface for Interactive Query Reformulation

ABSTRACT

Computer-readable media, computer systems, and computing methods are provided for classifying search results as either of good quality or of poor quality. Initially, a portion of the search results, such as the highest ranked documents, are selected for evaluation. A level of quality for each of the selected search results is determined using a classification process that includes the following steps: targeting features demonstrated by the selected search results to be evaluated; evaluating the selected features to generate a level-of-quality score for each of the selected search results; comparing the score against a predefined threshold value; and, based on the comparison, assigning each of the selected search results an absolute measurement. The absolute measurement indicates poor quality when the score is less than the threshold value. Upon recognizing that the selected search results are of poor quality, a corrective action that reformulates the issued search query is automatically invoked.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. application No. ______ (AttorneyDocket Number MS#331078.01/MFCP.157702) filed Dec. 15, 2010, entitled“CLASSIFYING RESULTS OF SEARCH QUERIES,” the teachings of which arehereby incorporated by reference in their entirety.

BACKGROUND

Search engines are generally designed with the intent to simplify users'lives by helping them find relevant search results (e.g., documents)that match their search queries in a fast and effective way. Forinstance, users of search engines may compose and submit search queriesin order conduct an online search for a service, product, particularinformation, or any other data targeted by the user intent behind thesearch query. Often, the set of search results generated by theconventional search engines are of poor quality, meaning that at least aportion of the set of search results does not properly satisfy theuser's intent behind the search query. Specifically, in one study,researchers discovered that the top-listed results in roughly one in tensets of search results were irrelevant to the user's intent of thecorresponding search query.

There are various reasons for generating poor-quality search results.Some reasons involve a lack of good-quality documents in a web indexbeing interrogated by the search engine or a failure of a core-rankingalgorithm to locate the good-quality documents that exist within the webindex.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Embodiments of the present invention address various reasons forgenerating poor-quality search results including the following: pooruser composition of search queries in a format that matches poor-qualitydocuments or other documents that do not satisfy the user's intentbehind the search query. These defectively composed search queries mayeither include extremely common terms that can be interpreted in amultitude of different ways (generating off-topic search results), orinclude rare terms that do not match any return meaningful documents(generating obscure search results).

Yet, search engines return these off-topic or obscure search results tothe user regardless of whether they are actually relevant to the user'sintent. This gives the user the false impression that, because thesearch results matched their search query, the returned search resultsmust be relevant. Upon inspection, however, users often find that thesearch results are not relevant. Further, if the user desires better,good-quality documents (used interchangeably with the phrase “searchresults” herein), the user is forced to recompose their search querywithout any assistance. This deficiency of search engine results in aninefficient and confusing user search experience.

Accordingly, because search engines fail to recognize that poor-qualitysearch results are being offered to the user as a result of the user'sdefectively composed search query, a novel search engine thatautomatically determines when the returned search results are of poorquality and, when poor-quality search results are determined, takescorrective action to improve the search query (i.e., re-articulating theuser's intent in the context of online searching) would provide anenhanced overall user experience.

Embodiments of the present invention address the problematic issue thatmany users do not possess proficient search-query formulation skills.For instance, users often type over-extensive or under-inclusive searchqueries and, upon issuing the search queries to a search engine, usersreceive search results that are irrelevant to the users' true intent.Other embodiments address the problematic issue that some search queriesare inherently vague (e.g., using common terms) or difficult to satisfy(e.g., using rare terms) regardless of how they are formulated.

Accordingly, embodiments of the present invention introduce variousmethods for detecting irrelevant search results and assisting a user inimproving search-result relevance. In one instance, the detection ofirrelevant search results is performed by a search engine that attemptsto build a classification process for automatically determining a levelof quality of search results within a search-engine results page (SERP)for a given user-initiated search query at runtime. In an exemplaryembodiment, the search engine implements the classification process toascertain whether the search results are acceptable in an absolute sense(e.g., either good quality or poor quality) based, in part, on thedetermined level of quality. Further, when these search results aredetermined to be of poor quality, they are labeled as such so that oneor more corrective actions may be carried out, if appropriate.

Because alterations to a user's search query, such as minor tweaks toterms therein, often improve the relevance of the returned searchresults, one of the corrective actions may involve implicitly rewritingthe search query on the user's behalf. In cases where there is highconfidence that the alterations will improve the search results, therewritten search query is prepared and submitted by the search engineautomatically in a transparent manner. In cases where there is a lowconfidence that the alterations will improve the search results, optionsfor altering key terms (e.g., remove or replace) are explicitlypresented to the user to solicit his or her feedback for reformulatingthe search query. In addition to helping the user reformulate his or hersearch query, another corrective action may include presenting searchresults identified to be of good quality, while suppressing those ofpoor quality. Accordingly, the user is spared the time involved inperusing through poor-quality search results.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in detail below withreference to the attached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing device suitable forimplementing embodiments of the invention;

FIG. 2 is a schematic diagram depicting a first illustrative UI displaythat includes a reformulation UI for altering key terms within a searchquery, in accordance with embodiments of the invention;

FIG. 3 is a block diagram of an exemplary network environment suitablefor use in implementing embodiments of the invention;

FIG. 4 is a schematic diagram depicting a second illustrative UI displaythat includes search results of poor quality that are generated by asearch engine, in accordance with embodiments of the invention;

FIG. 5 is a schematic diagram depicting a third illustrative UI displaythat includes the reformulation UI for improving a relevance of thesearch results of FIG. 4, in accordance with embodiments of theinvention;

FIG. 6 is a schematic diagram depicting a fourth illustrative UI displaythat includes the reformulation UI that exposes various presetsuggestions for altering key terms of a search query, in accordance withembodiments of the invention;

FIG. 7 is a flow diagram showing an overall method for invoking areformulation UI upon determining that a set of search results are oflow quality, in accordance with embodiments of the invention; and

FIG. 8 is a flow diagram showing an overall method for selecting acorrective action in response to low-quality search results, inaccordance with embodiments of the invention.

DETAILED DESCRIPTION

The subject matter of embodiments of the invention disclosed herein isdescribed with specificity to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies.

As can be gleaned from the disclosure below, various embodiments of thepresent invention pertain to determining a level of quality of searchresults and, potentially, taking a corrective action to improve thedetermined level of quality utilizing task-based engines (e.g., decisionengines, task engines, individual applications or operations, appletsystems, operating systems, and task-based mobile systems), or generalsystems that attempt to match search results to a user intent inherentwithin an issued search query. For simplicity of discussion, theseengines and/or systems will be hereinafter referred to as “searchengines.”

Embodiments of the invention described herein include computer-readablemedia having computer-executable instructions embodied thereon. Whenexecuted (e.g., using a processor), the computer-executable instructionsperform a method for invoking a reformulation user interface (UI) upondetermining that a set of search results are of low quality. Inembodiments, the method includes the steps of receiving a search queryissued by a user and returning the set of search results generated basedupon content of the search query. The method may further involvedetermining that a portion of the set of search results are of poorquality upon applying a classifying process thereto.

In embodiments, key terms are identified within the search query. Often,the key terms are identified by ignoring non-critical, common naturallanguage from the search query. Upon determining that a set of searchresults are of low quality, the search engine may render thereformulation UI to expose to the user preset suggestions in associationwith each of the key terms. When at least one user-initiated selectionis detected that targets one or more of the preset suggestions, thesearch query may be altered in accordance with the user-initiatedselection on the targeted preset suggestion(s).

In a second illustrative embodiment, the present invention introduces acomputer system for invoking a reformulation user interface (UI). In oneinstance, the computer system includes a processing unit coupled to acomputer-storage medium, where the computer storage medium stores aplurality of computer software components executable by the processingunit. As illustrated in FIG. 3, the computer system includes at leastthe computer software components comprising: a query-receivingcomponent; a classifying component; and an application component.

The query-receiving component is configured to receive a search queryissued by a user and to return the set of search results generated basedupon the search query. In other embodiments, the classifying componentis adapted to identify the query-receiving component returned searchresult(s) of poor quality within the set of search results. Generally,the application component is configured to invoke an appropriatecorrective action upon detecting the poor-quality search result(s). Inan exemplary embodiment, the appropriate corrective action is selectedbased on, in part, a level of confidence in an automatically rewrittensearch query producing new search results that are more relevant to theuser's intent of the issued search query than the returned set of searchresults. In one instance, the appropriate corrective action comprisesrendering the reformulation UI when the level of confidence isdetermined to be low. In another instance, the appropriate correctiveaction comprises issuing the automatically rewritten search query togenerate the new set of search results when the level of confidence isdetermined to be high.

In a third illustrative embodiment, a computerized method carried out bya search engine (running on a processor) for selecting a correctiveaction in response to low-quality search results. Initially, the methodincludes providing a set of search results returned in response to asearch query issued by a user of the search engine. The search enginemay target one or more of the set of search results that are ranked ashighly relevant to the issued search query. In embodiments, adetermination that the targeted search result(s) demonstrate a low levelof quality is made. Typically, the determination of a low level ofquality is made based on a relevance of the targeted search result(s) tothe issued search query.

In an exemplary embodiment, the search engine is responsible forestimating a high or low confidence, where a level of the confidence isbased upon whether a rewritten search query will generate new searchresults that are more relevant to the issued query than the returned setof search results. The rewritten search query, in embodiments, isautomatically generated by the search engine upon determining that thetargeted search result(s) demonstrate a low level of quality. In oneinstance, when a high confidence is estimated, the search engineautomatically issues the rewritten search query to generate the newsearch results. In another instance, when a low confidence is estimated,the search engine launches an interactive query refinement tool thatassists the user in altering key terms of the issued search query.

Generally, as discussed more fully below, the interactive queryrefinement tool is responsible for implementing actions. Some exemplaryactions include identifying one or more replacement terms for each ofthe key terms of the search query and presenting the replacement term(s)on a search-results page in proximity to the key terms. In operation ofthe tool, at least one key term within the issued search query isreplaced by the selected replacement term(s) upon detecting auser-initiated selection directed toward the presented replacementterm(s).

Having described an overview of embodiments of the present invention, anexemplary operating environment in which embodiments of the presentinvention may be implemented is described below in order to provide ageneral context for various aspects of the present invention.

Referring initially to FIG. 1 in particular, an exemplary operatingenvironment for implementing embodiments of the present invention isshown and designated generally as computing device 100. Computing device100 is but one example of a suitable computing environment and is notintended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing device 100be interpreted as having any dependency or requirement relating to anyone or combination of components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 1, computing device 100 includes a bus 110 thatdirectly or indirectly couples the following devices: memory 112, one ormore processors 114, one or more presentation components 116,input/output (I/O) ports 118, input/output components 120, and anillustrative power supply 122. Bus 110 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 1 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Theinventors recognize that such is the nature of the art, and reiteratethat the diagram of FIG. 1 is merely illustrative of an exemplarycomputing device that can be used in connection with one or moreembodiments of the present invention. Distinction is not made betweensuch categories as “workstation,” “server,” “laptop,” “hand-helddevice,” etc., as all are contemplated within the scope of FIG. 1 andreference to “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 100. In embodiments, thecomputer storage media represents tangible computer readable media, suchas medium embodied on a processor.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia. In embodiments, the communication media represents non-transitorycomputer readable media.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 100includes one or more processors that read data from various entitiessuch as memory 112 or I/O components 120. Presentation component(s) 116present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 118 allow computing device 100 to be logically coupled toother devices including I/O components 120, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

Turning now to FIG. 2, a schematic diagram depicting a firstillustrative UI display 200 that includes a reformulation UI foraltering key terms within a search query, in accordance with embodimentsof the invention. Initially, a search query 202 of “Microsoft SVC6Address” is entered into a query-receiving area 201 by a user. Thesearch query 202 is then employed to generate search results 205. Uponthe search engine evaluating the search results returned from an issuedsearch query, the search engine may determine that the search results205 are of poor quality (i.e., irrelevant to a user intent of the searchquery 202).

When the search engine determines the search results 205 are of poorquality (e.g., using a classification process described below), one ormore corrective actions are invoked. As depicted in FIG. 2, thecorrective actions that are invoked include presenting on asearch-engine results page (SERP) a portion of the search results 205while abstaining from showing a remainder of the set of search results,rendering a reformulation UI 210 that allows the user to alter key terms220 of the search query 202, and presenting an altered search query area250 that surfaces an altered search query 251. Typically, the alteredsearch query 251 reflects alterations made to the search query 202 upondetecting the user-initiated selection on one or more preset suggestions240.

In one instance, a group of preset suggests 222 for the search-queryterm “SVC6” may include a drop option 230. A user-initiated selection atthe drop option 230 will remove the term “SVC6” from the search query202. In another instance, a group of preset suggestions 222 for thesearch-query term “SVC6” may include a replacement term 223. Auser-initiated selection at the replacement term 223 will replace theterm “SVC6” within the search query 202 with an expanded meaning of theacronym “Silicon Valley Campus.” The user-initiated selection isrevealed by modifying a selectable control 221 (e.g., checking a checkbox or highlighting a dial control) proximate to the replacement term223. In another embodiment, the replacement term 223 is incorporatedinto the altered search query 251, as illustrated in FIG. 2.

Accordingly, the reformulation UI 210 acts as rich user-interactivedisplay that allows a user to dynamically modify the search query 202.In one instance, the reformulation UI 210 generates and displays thepreset suggestions 240, as discussed above. Although replacement terms223 and drop options 230 are illustrated in FIG. 2, many other presetsuggestions 240 may be offered to the user. By way of example, an addoption may be offered within the preset suggestions 240 that allows auser to add a term to the altered search query 251, which the searchengine provides for clarification of the search query 202.

In embodiments, the replacement terms 223 of the preset suggestions 240are selected by the search engine based on various factors. Initially,the search engine may dynamically access an alteration service (notshown) to inspect a group of terms that may serve as suitablereplacement terms (e.g., synonyms within the context of an onlinesearch) for the key terms 220 of the search query 202. The alterationservice generally performs an offline process that identifies a group ofterms as candidates for the replacement terms. Often, the alterationservice compiles these candidates based on previous experience (e.g.,user behavior extracted from query logs). By way of example, thealteration service monitors user behavior to create a path between termsthat are deemed to be interchangeable or refinements of one another.

In the instance of multi-user social activity, the alteration servicemay observe various users refining a search by switching out aparticular term. This switched-out term and its replacement may belinked by a path indicating they are synonymous with each other. Or, thealteration service may recognize that similar popular searches have oneterm replaced by another. For example, the alteration service may noticethat some users commonly issue a search query for “desktop computersales,” while other users commonly issue a search query for “laptopcomputer sales.” The alteration service, upon determining that similarsearch results are generated for both search queries immediately above,may identify “laptop” as a suitable replacement for “desktop” within thecontext of computers. Upon identifying terms as suitable replacements,the alteration service may draw a path between the terms to designatethem as replacements. As such, the “path” links the related terms in astorage location for eventual selection by the reformulation UI 210 ofFIG. 2.

In the instance of individual user activity, the alteration service maytrack a single user's online activity over a single session (e.g.,searching over discrete period of time). For instance, during a singlesession, when the user issues a search query that includes the term“desktop” and then refines the search query to include “laptop” in theplace of “desktop,” the alteration service learns that one is a validreformation of the other term. Consequently, the alteration service maylink the terms “laptop” and “desktop” via a path.

Upon linking related terms together via paths, the alteration servicemay organize related terms into groupings that facilitate subsequentaccess by the search engine. In an exemplary embodiment, the alterationservice may attach a confidence score to each of the terms assembledwithin a particular grouping. As used herein, the phrase “confidencescore” generally represents a level of interchangeability of the termswithin the groupings. For instance, a confidence score may be based on anumber of users that have interchanged the terms on a term-level, asopposed to a query-level. Generally, the greater the number of users, orthe greater the frequency of replacement, the higher the confidencescore. In operation, the confidence scores are employed to selectreplacement terms 223 from potential candidate terms within a groupingassociated with a key term of the search query 202. Often, thosecandidates that exhibit the highest-level confidence scores are revealedby the reformulation UI 210 as replacement terms. Advantageously, thealteration service provides the user with the most commonly taken actionby online users or the most useful action to the user, based on theuser's online history.

The system architecture for implementing the method of making anabsolute measurement of at least one search result and taking one ormore corrective actions will now be discussed with reference to FIG. 3.Initially, FIG. 3 is a block diagram illustrating a distributedcomputing environment 300 suitable for use in implementing embodimentsof the present invention. The exemplary computing environment 300includes a user device 310, a data store 335, a web server 350, a mobiledevice 370, and a network 380 that interconnects each of these items.Each of the user device 310, the data store 335, the web server 350, andthe mobile device 370, shown in FIG. 3, may take the form of varioustypes of computing devices, such as, for example, the computing device100 described above with reference to FIG. 1. By way of example only andnot limitation, the user device 310, the web server 350, and/or themobile device 370 may be a personal computer, desktop computer, laptopcomputer, consumer electronic device, handheld device (e.g., personaldigital assistant), various servers, processing equipment, and the like.It should be noted, however, that the invention is not limited toimplementation on such computing devices but may be implemented on anyof a variety of different types of computing devices within the scope ofembodiments of the present invention.

Typically, each of the user device 310, the web server 350, and themobile device 370 includes, or is linked to, some form of a computingunit (e.g., central processing unit, microprocessor, etc.) to supportoperations of the component(s) running thereon (e.g., query-receivingcomponent 351, classifying component 352, assignment component 353,application component 354, and the like). As utilized herein, the phrase“computing unit” generally refers to a dedicated computing device withprocessing power and storage memory, which supports operating softwarethat underlies the execution of software, applications, and computerprograms thereon. In one instance, the computing unit is configured withtangible hardware elements, or machines, that are integral, or operablycoupled, to the user device 310, the web server 350, and/or the mobiledevice 370, respectively, in order to enable each device to performcommunication-related processes and other operations (e.g., renderingthe reformulation UI 210 of FIG. 2). In another instance, the computingunit may encompass processor(s) (not shown) coupled to thecomputer-readable medium accommodated by each of user device 310, theweb server 350, and the mobile device 370.

Generally, the computer-readable medium includes physical memory thatstores, at least temporarily, a plurality of computer softwarecomponents that are executable by the processor. As utilized herein, theterm “processor” is not meant to be limiting and may encompass anyelements of the computing unit that act in a computational capacity. Insuch capacity, the processor may be configured as a tangible articlethat processes instructions. In an exemplary embodiment, processing mayinvolve fetching, decoding/interpreting, executing, and writing backinstructions.

Also, beyond processing instructions, the processor may transferinformation to and from other resources that are integral to, ordisposed on, the user device 310, the web server 350, and the mobiledevice 370. Generally, resources refer to software components orhardware mechanisms that enable the user device 310, the web server 350,and the mobile device 370 to perform a particular function. By way ofexample only, resource(s) accommodated by the web server 350 operate toassist the search engine 345 in receiving inputs from a user at the userdevice 310 or the mobile device 370 and/or providing an appropriatecommunication in response to the inputs (e.g., rendering search results325).

The user device 310 may include an input device (not shown) and apresentation device 315. Generally, the input device is provided toreceive input(s) affecting, among other things, search results 325rendered by a search engine 345 and surfaced at a UI display 320.Illustrative input devices include a mouse, joystick, key pad,microphone, I/O components 120 of FIG. 1, or any other component capableof receiving a user input and communicating an indication of that inputto the user device 310. By way of example only, the input devicefacilitates entry of a search query 375, which is communicated over thenetwork 380 to the web server 350 for processing by the search engine345.

In embodiments, the presentation device 315 is configured to renderand/or present the UI display 320 thereon. The presentation device 315,which is operably coupled to an output of the user device 310, may beconfigured as any presentation component that is capable of presentinginformation to a user, such as a digital monitor, electronic displaypanel, touch-screen, analog set-top box, plasma screen, audio speakers,Braille pad, and the like. In one exemplary embodiment, the presentationdevice 315 is configured to present rich content, such as the searchresults 325, associated advertisements, and digital images and videos.In another exemplary embodiment, the presentation device 315 is capableof rendering other forms of media (i.e., audio signals).

The data store 335 is generally configured to store informationassociated with selecting preset suggestions for presentation in areformulation UI. In operation, when the user is implementing a searchat the search engine 345, the search engine 345 may access theinformation residing at the data store 335, such as candidatereplacement terms for one or more of the key terms within a searchquery.

In other embodiments, the data store 335 may be configured to besearchable for suitable access to the information maintained thereby.For instance, the data store 335 may be searchable for preset options.It will be understood and appreciated by those of ordinary skill in theart that the information stored in the data store 335 may beconfigurable and may include any information relevant to thedetermination and/or the retrieval of the a search-result level ofquality, as well as the selection of the corrective actions that aretriggered therefrom. The content and volume of such user-interactioninformation are not intended to limit the scope of embodiments of thepresent invention in any way. Further, though illustrated as single,independent components, the data store 335 may, in fact, be a pluralityof databases, for instance, a database cluster, portions of which mayreside on the user device 310, the web server 350, the mobile device370, another external computing device (not shown), and/or anycombination thereof.

This distributed computing environment 300 is but one example of asuitable environment that may be implemented to carry out aspects of thepresent invention and is not intended to suggest any limitation as tothe scope of use or functionality of the invention. Neither should theillustrated distributed computing environment 300 be interpreted ashaving any dependency or requirement relating to any one or combinationof the devices 310, 350, and 370, the data store 335, and components351-354 as illustrated. In some embodiments, one or more of thecomponents 351-354 may be implemented as stand-alone devices. In otherembodiments, one or more of the components 351-354 may be integrateddirectly into the web server 350, or on distributed nodes thatinterconnect to form the web server 350. It will be appreciated andunderstood that the components 351-354 (illustrated in FIG. 3) areexemplary in nature and in number and should not be construed aslimiting.

Accordingly, any number of components may be employed to achieve thedesired functionality within the scope of embodiments of the presentinvention. Although the various components of FIG. 3 are shown withlines for the sake of clarity, in reality, delineating variouscomponents is not so clear, and, metaphorically, the lines would moreaccurately be grey or fuzzy. Further, although some components of FIG. 3are depicted as single blocks, the depictions are exemplary in natureand in number and are not to be construed as limiting (e.g., althoughonly one presentation device 315 is shown, many more may becommunicatively coupled to the user device 310).

Further, the devices of the exemplary system architecture may beinterconnected by any method known in the relevant field. For instance,the user device 310, the web server 350, and the mobile device 370 maybe operably coupled via a distributed computing environment thatincludes multiple computing devices coupled with one another via one ormore networks (not shown). In embodiments, the network may include,without limitation, one or more local area networks (LANs) and/or widearea networks (WANs). Such networking environments are commonplace inoffices, enterprise-wide computer networks, intranets, and the Internet.Accordingly, the network is not further described herein.

In operation, the components 351-354 are designed to perform a processthat includes, at least, taking corrective actions when the searchresults are deemed to be of poor quality. Initially, the search engine345 includes the query-receiving component 351 configured to receive thesearch query 375 issued by a user and to return the set of searchresults 325 that are generated, in part, based upon the search query375. In this way, the query-receiving component 351 acts as a frontendmechanism that interfaces with a user, while the backend mechanismretrieves the search results 325 from a web index.

The classifying component 352 is configured to carry out theclassification process that automatically determines a qualitymeasurement of one or more search results of the set of search results325. Generally, the classification process is employed to automaticallydetermine whether the search results 325 are acceptable (i.e., satisfythe user intent of the search query 375) in an absolute sense.Typically, the classification process includes the steps of evaluatingvarious features exhibited by the search result(s) 325 to determine anoverall level of quality and deriving the absolute measurement from theoverall level of quality determined for each of the search result(s)325. Generally, the absolute measurement identifies the search result(s)325, respectively, as either of good quality or of poor quality in amutually exclusive manner.

In an exemplary embodiment, the assigning component 353 is configured tolabel each of the search result(s) 325 as either “good quality” or “poorquality” based on an outcome of the classification process. Theselabeled search results are passed to the application component 354, thatinvokes an appropriate corrective action when one or more of the searchresults (e.g., the X-number of highest ranked search results or amajority of the search results on the SERP) are labeled as “poorquality.”

In an exemplary embodiment, the appropriate corrective action isselected as a function of a level of confidence. As used herein, the“level of confidence” is not meant to be limiting in definition, butencompasses any metric that relates to a likelihood that any alternativeterms within a rewritten search query, offered by the applicationcomponent 354, will produce new search results that are improved overthe current “poor quality” search results 325. Concisely, inembodiments, the level of confidence indicates a probability that asearch-engine composed search query will produce enhanced searchresults. In application, the level of confidence is used to select oneor more corrective actions, such as rendering a reformulation userinterface (UI) 340, alerting a user that the search result(s) 325 arelabeled “poor quality” via a notification message, abstaining frompresenting to the user the poor-quality search results, or automaticallyrewriting the search query 375 to generate an updated set of searchresults.

In an exemplary embodiment, the search query 375 is automaticallyrewritten and automatically issued upon calculating a high level ofconfidence in the alternative terms suggested by the applicationcomponent 354. In contrast, the reformulation UI 340 is rendered uponcalculating a low level of confidence in the alternative terms suggestedby the application component 354. In one instance, the reformation UI340 solicits and accepts user-initiated interactions 385, such asmouse-click selections, for guiding the alteration of key terms withinthe search query. By way of example, altering key terms may involvereplacing or removing one or more key terms within the search query 375.

Although various different methods enhancing the search results 325 upondetermining that one or more search results are labeled “poor quality”have been described, it should be understood and appreciated that othertypes of suitable methods and interfaces for selecting and implementingcorrective actions may be used, and that embodiments of the presentinvention are not limited to those corrective actions discussed above.For instance, an additional corrective action may involve presenting athird-party entity the opportunity to reformulate the search query 375using the third-party entity's experience and resources.

Turning now to FIG. 4, a schematic diagram depicting a secondillustrative UI display 400 that includes search results 415 of poorquality that are generated by a search engine is shown, in accordancewith embodiments of the invention. Initially, the search query“Carbright in Spokane” 410 is entered within the query-entry area 201.As illustrated, the matching terms “bright cloth” 425 within the searchresults 425 do not appear be relevant to the user's intent of the searchquery 410. Accordingly, the search engine may determine that the searchresults 415 are of poor quality and invoke one or more correctiveactions.

With reference to FIG. 5, a schematic diagram depicting a thirdillustrative UI display 500 that includes a reformulation UI 530 forimproving a relevance of the search results 415 of FIG. 4 is shown, inaccordance with embodiments of the invention. In embodiments, thereformulation UI 530 lists the key terms 220 of the search query 410. Asillustrated, only one of the key terms “Carbright” 551 is provided withpreset suggestions 550. These preset suggestions 550 are each pairedwith a selectable control proximate thereto. For instance, thereplacement term “Carbrite” of the preset suggestions 550 is displayedin association with a selectable control 540 configured as a check box.In operation, a user-initiated selection targeting the selectablecontrol 540 may incorporate replacement term “Carbrite” within analtered search query 511 by replacing the key word “Carbright” 551.

Beyond the reformulation UI 530, other corrective actions are taken. Forinstance, irrelevant search results 415 are removed from the SERP, incomparison to FIG. 4. Also, a notification message 520 is displayedalerting a user that certain query results 415 were deemed irrelevantand, thus, suppressed.

With reference to FIG. 6, a schematic diagram depicting a fourthillustrative UI display 600 that includes a reformulation UI 610 thatexposes various preset suggestions 640 for altering key terms of asearch query 605, in order to improve the search results 620, is shown,in accordance with embodiments of the invention. In this exemplaryreformulation UI 610, replacement terms 635 are offered within thepreset suggestions 640, as well as replacement search queries 630. Inoperation, a user-initiated selection of one of the replacement searchqueries 630 will change the search query 605 to an altered search querythat matches the selected replacement query. Typically, the replacementsearch queries 630 are generated by the alteration service in a similarfashion as the replacement terms within the preset suggestions 640, asdiscussed more fully above.

As shown, with respect to the key term “Chips” of the search query 605,the preset suggestions 640 include replacement terms/phrases (“Chip” and“Poker Chip”) and an option to remove the key term (“Chips”). In theembodiment shown in FIG. 6, the replacement terms 640 are presentedwithout selectable controls, such as check boxes or radio buttons.Instead, a user may select a desired preset suggestion 640 by selectinga representation (e.g., text) of one or more preset suggestions 640directly. In one instance, a highlight element 635 (e.g., bold text,highlighting, border around text, and the like) may appear inassociation with the particular preset suggestion that the user's cursoroverlays, thereby notifying the user which of the preset suggestions 640will be invoked upon the user performing a selection action.

It should be noted that the replacement phrase “Poker Chip,” which isdistinguished by the highlight element 635, includes the term “Poker,”which was not originally present within the search query 605. Further,the term “Poker” is neither a synonym (e.g., “Show” to “View” or “See”)a similar descriptor (e.g., “Blue” to “Green” or “Red”), nor anexpansion/truncation/variation (e.g., “Chips” to “Chip”) of a termwithin the search query 605. Instead, the term “Poker” is a word derivedby the alteration service that frequently appears in documents thatinclude one or more of the key terms. In an exemplary embodiment, thenewly derived term “Poker” is paired with the key term (“Chips”) in thesearch query 605 in which the derived term most commonly appears inproximity with. For instance, the term “Poker” may be initially selectedfor being a most prevalent key term within a group of search resultsthat also include key terms from the search query 605. However, becausethe phrase “Poker Chip” is more common than any other combination of“Poker” and a key term in the search query 605, the derived term “Poker”is paired with the key term “Chips” within the preset suggestions 640.

With continued reference to FIG. 6, the “classification process” fordetermining when the reformulation UI 610, or any other correctiveaction, is presented will now be discussed. Generally, upon accessing aset of search results 620 returned from the issued search query 605, thesearch engine determines a level of quality of a portion of the completeset of search results 620 using the classification process. Initially,the classification process may involve applying various classifyingcriteria to one or more search results individually. In instances, thephrase “classifying criteria” broadly pertains to parameters forevaluating any feature demonstrated by the search results that can beexamined. By way of example, some of the features demonstrated by thesearch results that may be evaluated are ranking features, quality-levelfeatures, caption-level features, session-level features, and aggregatefeatures. In operation, one or more of these features may be targetedfor evaluation and the evaluations of these feature(s) are aggregated togenerate a level-of-quality score for each of the search results.

Initially, the ranking features represent those features that serve tomeasure a relationship between key terms within the search query 605 andcontents of the set of search results 620 (e.g., documents, web page,blogs, and the like). In an exemplary embodiment, the ranking featurespertain to query-dependent features of the search results 620, such asnumber of matches between the key terms of the search query 605 andcontent of the search results 620. Often, one or more of the rankingfeatures are employed for the task of ranking the search results withrespect to one another, in order to develop a search-result order on theSERP.

In one example, the ranking features may include a number/frequency ofmatches between terms in the search query and the title of the searchresult, anchor text pointing to the search result, body of the searchresult, and/or critical sections (e.g., top of page) of the searchresult. In embodiments, not all terms of the search query areconsidered. Typically, the search engine is concerned with whether thesearch results match, in some way, to the linguistic head (e.g., keyterms, rare terms, and the like) of the search query. Accordingly, inembodiments, the linguistic head may be distilled from the search queryand employed to evaluate ranking features of the search results, whilenon-critical, common, natural language of the search query may beignored for purposes of classification.

In another example, the ranking features pertain to a proximity ofmatched key terms within the search results. Typically, the closertogether the matched terms appear within the search results, the morerelevant the search results are with respect to the search query, thus,warranting a high level of quality designation. As discussed throughout,a search result designated as having a high level of quality is usuallyassigned a high score within the absolute scale, which often translatesinto an absolute measurement of “good quality.”

The quality-level features generally represent features thatindividually quantify an overall quality of the set of search resultsindependent of the search query. In embodiments, the overall quality isascertained upon inspecting the contents/title/anchor text of a searchresult and/or a domain name integral to the search result. In oneexample, inspecting the contents of the search results involvesenumerating a number of misspellings within a body of a document, wherea large number of misspellings typically decreases a level of qualityassociated with the search results. In another example, inspecting thesearch result's domain involves determining whether the search resultnavigates to a trusted web site. As can be seen, the quality-levelfeatures focus on attributes of the search results independent of termswithin the search query.

The caption-level features, in one instance, represent features thatenumerate a number of matches between key terms within the search queryand a search-engine prepared summary (i.e., snippet or instance answer)that is particular to each of set of search results. In anotherinstance, the caption-level features pertain to a determination ofwhether the snippets are well formed (denoting a high-quality searchresults), or whether the snippets include incomplete sentences, AmericanStandard Code for Information Interchange (ASCII) junk material, amultitude of numerals/symbols/uniform resource locators (URLs), and/orother non-textual data (denoting a poor-quality search result).

The session-level features generally represent features that labelsearch results within the set of search results as either relevant orirrelevant based upon user-initiated refinements to the search resultsduring a course of an online session. In one instance, the refinementspertain to a number of times the user revised the search query duringthe online session. Often, more refinements to the search query signifythat the user is dissatisfied with the returned search results, whichequates to lower levels of quality assigned to the search resultspresented in response to the search query. In another instance, therefinements pertain to an amount of user-interactions (e.g., selections)directed toward the search results that are generated based on thesearch query. Often, when little to no user-initiated selections (e.g.,mouse click actions) are directed toward the search results, users areuninterested in the search results, thereby indicating that the searchresults are irrelevant or unpopular in some fashion. Generally, searchresults found to be unpopular are designated as having a low level ofquality.

The aggregate features generally represent features that considervariation between contents of the set of search results, advertisementspublished at the search-results page, and/or terms comprising relatedsearches offered by the search engine in tandem with the set of searchresults. As a general rule, in embodiments, high-quality search resultsare search results that exhibit a sufficient amount of inter-diversitywith one another without being overly varied. In one instance, thisinter-diversity is articulated as cross-result entropy. As used herein,“cross-result” entropy refers to the randomness, or inconsistencies,expressed by the constituents of the SERP. In one example, where thesearch results are similar (e.g., sharing one or two domains and/orexposing common subject matter within their content), the cross-resultentropy is very low. This low cross-result entropy may occur upon theuser entering an over-specified search query, which decreases thesearch-result level of quality because some inter-diversity is helpfulto the user. In another example, where the search results are entirelydistinct (e.g., encompassing a wide variety of domains and/or preventingclustering of search results due to lack of common subject matter withintheir content), the cross-result entropy is very high. This highcross-result entropy may occur upon the user entering an under-specifiedsearch query, which also decreases the search-result level of qualitybecause too much inter-diversity is confusing to the user.

Although the content and domains associated with the search results havebeen described, it should be understood and appreciated that any othersuitable attributes of the search results or suitable information on theSERP may be employed to calculate or influence the cross-result entropy,and embodiments of the present invention are not limited to theattributes described above. For instance, entropy between advertisementsand/or related-searches data may be considered in adjusting a level ofquality of the search results.

In a hypothetical example, the above-referenced features will now bediscussed in relation to a search query 605 for “SHOW HOW MUCH BLUEDIAMOND CHIPS ARE WORTH,” where the user's intent of the search query605 is to determine a value of “Blue Diamond” poker chips used forgambling at a particular casino. In some instances, as shown in FIG. 6,the search engine may distill the linguistic head from the search query605 such that the key terms “blue,” “diamond,” “chips,” and “worth” areused in the classification process, while non-critical, common, naturallanguage of the search query (e.g., “show how much”) may be ignored forpurposes of classification. These distilled linguistic heads may beoffered as the replacement search queries 630 within the reformulationUI 610. As shown, some of the replacement search queries 630 not onlyrecommend removing all terms besides the key terms, but also offersuggested replacements for one or more of the key terms.

In other instances, the reformulation UI 610 prompts the user toindividually alter any term within the search query 605, regardless ofwhether the term is designated as a key term. Thus, the user is allowedto manually target one or more terms of the search query 605 foralteration based on his or her own user intent, without substantialautomated intervention. As discussed above, upon the user targeting andselecting a term to alter, the search query may be automatically updatedwith the altered term and transparently issued to the search engine,thereby dynamically generating a sampling of new search results in realtime, which correspond to the updated search query.

As illustrated in the exemplary UI 600 of FIG. 6, a top-ranked searchresult 625 of the displayed search results 620 may be a documententitled “Wholesale Loose Diamond Chips.” This example search result 625will be utilized within the discussion below to illustrate a functionalapplication of the classification process. Initially, the rankingfeatures of the example search result 625 may be applied to measure arelationship between the key terms within the search query 605 andcontents of the example search result 625. In an exemplary embodiment,there may be a high number of matches between the search-query terms“Diamond,” “Chips,” and “Worth” while very few matches to “Blue.”Specifically, these matches may occur within the title, which includes“Diamond Chips,” and may occur frequently within the content of thesearch results. (This may be why the example search result 625 was atop-ranked search result with respect to the other search results 620 onthe SERP.) As a result, the classification process may evaluate theexample search result 625 as relevant to the user intent of the searchquery 605 upon analysis of the ranking features. Accordingly, theranking features may identify the example search 625 result as a “goodquality” search result and an outcome of the evaluation of the rankingfeatures might be to incrementally raise the value of the score of theexample search result within the absolute scale.

In another example, the quality-level features of the example searchresult 625 may be applied to individually quantify an overall quality ofthe example search results independent of the search query 605. In thishypothetical example, the overall quality of the example search result625 is ascertained upon inspecting internal characteristics of theexample search result 625. In one instance, the content of the examplesearch result 625 may be inspected to enumerate a number of misspellingswithin a body of the document. Because the example search result 625 ismaintained by a diamond-chip dealer, a large number of misspellings maybe detected within the document's body (e.g., a comparatively greaterfrequency of misspellings with respect to an average web page). As aresult, the classification process may evaluate the example searchresult 625 as of poor quality upon analysis of the quality-levelfeatures. Accordingly, the quality-level features may identify theexample search result 625 as a “poor quality” search result and anoutcome of the evaluation of the quality-level features might be toincrementally decrease the value of the score of the example searchresult 625 within the absolute scale.

In another instance, with respect to the quality-level features, theexample search result's domain may be inspected to determine whether theexample search result 625 navigates to a trusted web site. Again,because the example search result 625 is maintained by a diamond-chipdealer, the address for the web site that hosts the example searchresult 625 will likely be an non-trusted site, as opposed to the website of a large, well-known, reputable company. As a result, theclassification process may again evaluate the example search result 625as having poor quality upon analysis of the quality-level features.Accordingly, the quality-level features may again identify the examplesearch result 625 as a “poor quality” search result and may againincrementally decrease the value of the score of the example searchresult within the absolute scale.

In yet another example, the caption-level features of the example searchresult 615 may be applied to enumerate a number of matches between thekey terms within the search query 605 and a search-engine preparedsnippet of the example search result that reads “DIAMOND BUYING IS BUYERBEWARE SELLING . . . YOU GET A PICTURE THAT MAY SHOW THE DIAMOND IN ITSBEST COLORS.” In this snippet, the key term “Diamond” is matched twice,the key term “Blue” relates to the snipped term “Colors,” and the keyterms “Chips” and “Worth” are not matched at all. As such, theclassification process may evaluate the example search result 615 ashaving a poor snippet upon analysis of the caption-level feature.

In still another example, the session-level features of the examplesearch result 615 may be applied to label the example search result 615as either relevant or irrelevant based upon user-initiated refinementsto the search results 620 during a course of an online session. In oneinstance, if the example search result 615 appeared in a SERP during thesame online session and, in response to viewing the example searchresult 625, the user refined the search query, the session-levelfeatures indicate that the user is dissatisfied with the example searchresult 615 and the example search result is of poor quality. In anotherinstance, if the example search result 615 appeared in a SERP during thesame online session, but received little to no user-initiated selections(e.g., mouse click actions), the session-level features indicate thatthe user found the example search result irrelevant the user's intent.

In a final example, the aggregate features of the example search result625 may be applied to consider variation between contents of the searchresults compared to the example search result 625. For instance, if allthe search results, including the example search result 625, relate todiamond dealers (exposing common subject matter within their content),the cross-result entropy is very low, which decreases the search-resultlevel of quality because some inter-diversity is helpful to the user.

Upon evaluating various features and/or other classifying criteria, theclassification process may make a prediction of the search result levelof quality. In an exemplary embodiment, the prediction is based on, inpart, which rating (good or poor) each of the classifying criteriaindicate upon evaluation. When a substantial number of the classifyingcriteria point in a common direction, the prediction is finalized. Forinstance, with reference to the hypothetical example above, theevaluation of the features above resulted in the following: rankingfeatures indicated good quality; quality-level features indicated a poorquality; caption-level features indicated at poor quality; session-levelfeatures indicated a poor quality; and aggregate features indicated apoor quality. Because, four of the five classes of analytical features,upon application to the example search result 625, indicate the examplesearch result 625 is of a poor quality, the prediction generated by theclassification process may classify the example search result 615 as alow level of quality, thereby potentially triggering corrective action(e.g., presenting the reformulation UI 610).

Turning now to FIG. 7, a flow diagram is shown depicting an illustrativemethod 700 for invoking a reformulation UI upon determining that a setof search results are of low quality, in accordance with embodiments ofthe invention. Initially, it should be appreciated and understood thatalthough the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

In an exemplary embodiment, the method 700 involves the steps ofreceiving a search query issued by a user (see block 710) and returningthe set of search results generated based upon content of the searchquery (see block 720). The method 700 may further involve determiningthat a portion of the set of search results are of poor quality uponapplying a classifying process thereto, as indicated at block 730.

In embodiments, as indicated at block 740, key terms are identifiedwithin the search query. Often, the key terms are identified by ignoringnon-critical, common natural language from the search query. Upondetermining that a set of search results are of low quality, the searchengine may render the reformulation UI to expose to the user presetsuggestions in association with each of the key terms, as indicated atblock 750. When at least one user-initiated selection is detected thattargets one or more of the preset suggestions (see block 760), thesearch query may be altered in accordance with the user-initiatedselection on the targeted preset suggestion(s) (see block 770).

Referring to FIG. 8, a flow diagram is shown depicting illustrativemethod 800 for selecting a corrective action in response to low-qualitysearch results, in accordance with embodiments of the invention.Initially, the method 800 may represent a computerized method carriedout by a search engine (running on a processor). In embodiments, themethod 800 includes providing a set of search results returned inresponse to a search query issued by a user of the search engine, asindicated at block 810. As indicated at block 820, the search engine maytarget one or more of the set of search results that are ranked ashighly relevant to the issued search query. In embodiments, adetermination that the targeted search result(s) demonstrate a low levelof quality is made, as indicated at block 830. Typically, thedetermination of a low level of quality is made based on a relevance ofthe targeted search result(s) to the issued search query.

In an exemplary embodiment, as indicated at block 840, the search engineis responsible for estimating a high or low confidence, where a level ofthe confidence is based upon whether a rewritten search query willgenerate new search results that are more relevant to the issued querythan the returned set of search results. The rewritten search query, inembodiments, is automatically generated by the search engine upondetermining that the targeted search result(s) demonstrate a low levelof quality. In one instance, as indicated at block 850, when a highconfidence is estimated, the search engine automatically issues therewritten search query to generate the new search results. In anotherinstance, as indicated at block 890 when a low confidence is estimated,the search engine launches an interactive query refinement tool thatassists the user in altering key terms of the issued search query.

Generally, as discussed more fully below, the interactive queryrefinement tool is responsible for implementing actions. Some exemplaryactions include identifying one or more replacement terms for each ofthe key terms of the search query (see block 860) and presenting thereplacement term(s) on a search-results page in proximity to the keyterms (see block 870). In operation of the tool, as indicated at block880, at least one key term within the issued search query is replaced bythe selected replacement term(s) upon detecting a user-initiatedselection directed toward the presented replacement term(s).

Various embodiments of the invention have been described to beillustrative rather than restrictive. Alternative embodiments willbecome apparent from time to time without departing from the scope ofembodiments of the inventions. It will be understood that certainfeatures and sub-combinations are of utility and may be employed withoutreference to other features and sub-combinations. This is contemplatedby and is within the scope of the claims.

1. One or more computer-readable media having embodied thereoncomputer-executable instructions that, when executed, perform a methodfor invoking a reformulation user interface (UI) upon determining that aset of search results are of low quality, the method comprising:receiving a search query issued by a user; returning the set of searchresults generated based upon content of the search query; determiningthat a portion of the set of search results are of poor quality uponapplying a classifying process thereto; identifying key terms within thesearch query; rendering the reformulation UI to expose to the userpreset suggestions in association with each of the key terms, whereinthe reformulation UI prompts the user to dynamically alter theidentified key terms individually; detecting at least one user-initiatedselection targeting one or more of the preset suggestions; and alteringthe search query in accordance with the at least one user-initiatedselection on the one or more targeted preset suggestions.
 2. The mediaof claim 1, wherein the classification process comprises: determining alevel of quality of the set of search results individually by applying aplurality of classifying criteria thereto; assigning at least one searchresult of the set of search results a score within an absolute scalebased upon the determined level of quality; and comparing the scoreagainst a threshold value within the absolute scale.
 3. The media ofclaim 2, wherein the classification process further comprises:classifying the at least one search result as being of good quality uponthe comparison revealing that the score is greater than the thresholdvalue; and classifying the at least one search result as being of poorquality upon the comparison revealing that the score is less than thethreshold value.
 4. The media of claim 1, wherein the method furthercomprises: ranking search results of the set of search results againstone another based on, in part, relevance to a user intent implicitwithin the search query; and targeting high-ranked search results withinthe ranking for receiving application of the classification process. 6.The media of claim 1, wherein identifying key terms within the searchquery comprises ignoring non-critical, common natural language from thesearch query.
 7. The media of claim 1, wherein rendering thereformulation UI comprises presenting a selectable control proximate toa respective preset suggestion.
 8. The media of claim 7, wherein theselectable control comprises at least one of a check box or a radiobutton.
 9. The media of claim 8, wherein detecting at least oneuser-initiated selection targeting one or more of the preset suggestionscomprises detecting the user-initiated selection upon the selectablecontrol proximate to a respective preset suggestion.
 10. The media ofclaim 1, wherein the preset suggestions include a drop option thatallows a user to remove a key term corresponding to the drop option fromthe search query.
 11. The media of claim 1, wherein the presetsuggestions include at least one replacement term that allows a user toreplace a key term corresponding to the replacement term inside thesearch query.
 12. The media of claim 1, wherein the preset suggestionsinclude an add option that allows a user to incorporate a new term tothe search query.
 13. The media of claim 1, wherein the method furthercomprises presenting on a search-results page the search query issued bythe user and an altered search query upon detecting the user-initiatedselection on one or more of the preset suggestions, wherein the alteredsearch query reflects alterations made to the user-issued search queryvia the reformulation UI.
 14. The media of claim 1, wherein the methodfurther comprises abstaining from showing the portion of the set ofsearch results upon determining that the portion of the set of searchresults are of poor quality.
 15. The media of claim 1, wherein themethod further comprises: upon altering the search query, automaticallygenerating a new set of search results using the altered search query;and dynamically presenting a preview of the new set of search results.16. A computer system for invoking a reformulation user interface (UI),the computer system comprising a processing unit coupled to acomputer-storage medium, the computer storage medium having storedthereon a plurality of computer software components executable by theprocessing unit, the computer software components comprising: aquery-receiving component to receive a search query issued by a user andto return the set of search results generated based upon the searchquery; a classifying component to identify the query-receiving componentreturned one or more search results of poor quality within the set ofsearch results; an application component to invoke an appropriatecorrective action upon detecting the one or more poor-quality searchresults, wherein the appropriate corrective action is selected based on,in part, a level of confidence in an automatically rewritten searchquery producing new search results that are more relevant to the user'sintent of the issued search query than the returned set of searchresults, wherein the appropriate corrective action comprises renderingthe reformulation UI when the level of confidence is determined to below, wherein the reformulation UI prompts the user to dynamically alterterms within the issued search query individually; and wherein theappropriate corrective action comprises issuing the automaticallyrewritten search query to generate the new set of search results whenthe level of confidence is determined to be high.
 17. A computerizedmethod carried out by a search engine running on a processor forselecting a corrective action in response to low-quality search results,the method comprising: providing a set of search results returned inresponse to a search query issued by a user of the search engine;targeting one or more search results of the set of search results thatare ranked as highly relevant to the issued search query; determiningthat the one or more targeted search results demonstrate a low level ofquality as a function of relevance to the issued search query;estimating a high or low confidence in a rewritten search querygenerating new search results that are more relevant to the issued querythan the returned set of search results, wherein the rewritten searchquery is automatically generated by the search engine upon determiningthat the one or more targeted search results demonstrate a low level ofquality; when a high confidence is estimated, automatically issuing therewritten search query to generate the new search results; and when alow confidence is estimated, launching an interactive query refinementtool that assists the user in altering key terms of the issued searchquery, wherein the interactive query refinement tool is responsible forimplementing actions comprising: (a) identifying one or more replacementterms for each of the key terms of the search query; (b) presenting theone or more replacement terms on a search-results page in proximity tothe key terms; and (c) upon detecting a user-initiated selectiondirected toward the one or more presented replacement terms, replacingat least one key term within the issued search query with the one ormore selected replacement terms.
 18. The computerized method of claim17, wherein identifying one or more replacement terms comprisesaccessing a group of candidate terms for each of the key terms of thesearch query, where the group of candidate terms are linked to arespective key term via a pre-established navigation path, and whereinthe pre-established navigation path is built by monitoring userinteractions with the search engine.
 19. The computerized method ofclaim 18, wherein the user interactions with the search engine compriseat least one of multi-user search behavior that indicates the group ofcandidate terms are interchangeable with a respective key term linkedthereto, or individual-user activity during a single search session thatindicates the group of candidate terms are employed to refine arespective key term linked thereto.
 20. The computerized method of claim19, wherein identifying one or more replacement terms comprises: rankingeach of the candidate terms within the group based, in part, on afrequency at which a particular candidate term is interchanged with oremployed to refine a respective key word; and promoting the candidateterms having a highest ranking within the group to the one or morereplacement terms.